System and Method for Identifying Organizational Elements in Argumentative or Persuasive Discourse

ABSTRACT

In accordance with the teachings described herein, systems and methods are provided for identifying organizational elements in argumentative or persuasive discourse. A text that has been annotated is received. The annotated text includes argumentative or persuasive discourse that includes claims and evidence and organizational elements configured to organize the claims and evidence. Annotations of the annotated text distinguish the organizational elements from the claims and evidence. A rule set or a feature set is identified from the annotated text, where the rule set or the feature set includes textual patterns or word frequency features related to the organizational elements of the annotated text. A model is built based on the annotations and on the rule set or the feature set. The model is configured to identify organizational elements in a new text. The model is applied to the new text.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Patent ApplicationNo. 61/583,445, filed Jan. 5, 2012, entitled “Identifying High-LevelOrganizational Elements in Argumentative Discourse,” which is hereinincorporated by reference in its entirety.

FIELD

The technology described in this patent document relates generally todetection of organizational elements in argumentative or persuasivediscourse and more particularly to applying a rule-based system or aprobabilistic sequence model for automated identification oforganizational elements in argumentative or persuasive discourse.

BACKGROUND

When presenting an argument or attempting to persuade an audience, awriter or speaker generally cannot simply state a list of claims andpieces of evidence. Rather, the speaker or writer must generallystructure the claims and pieces of evidence and explain how they relateto an opponent's argument. Thus, argumentative or persuasive discoursemay include not only language expressing the claims and evidence butalso language used to organize the claims and evidence.

SUMMARY

In accordance with the teachings described herein, systems and methodsare provided for identifying organizational elements in argumentative orpersuasive discourse. In a method for identifying organizationalelements in argumentative or persuasive discourse, a text that has beenannotated is received. The annotated text includes argumentative orpersuasive discourse that includes claims and evidence andorganizational elements configured to organize the claims and evidence.Annotations of the annotated text distinguish the organizationalelements from the claims and evidence. A rule set or a feature set isidentified from the annotated text, where the rule set or the featureset includes textual patterns or word frequency features related to theorganizational elements of the annotated text. A model is built based onthe annotations and on the rule set or the feature set. The model isconfigured to identify organizational elements in a new text. The modelis applied to the new text.

A system for identifying organizational elements in argumentative orpersuasive discourse includes a data processor and computer-readablememory in communication with the data processor encoded withinstructions for commanding the data processor to execute steps. Thesteps include receiving text that has been annotated, where theannotated text includes argumentative or persuasive discourse thatincludes claims and evidence and organizational elements configured toorganize the claims and evidence. Annotations of the annotated textdistinguish the organizational elements from the claims and evidence.The steps also include identifying a rule set or a feature set from theannotated text, where the rule set or the feature set include textualpatterns or word frequency features related to the organizationalelements of the annotated text. The steps further include building amodel based on the annotations and on the rule set or the feature set,where the model is configured to identify organizational elements in anew text. The model is applied to the new text.

A non-transitory computer-readable storage medium for identifyingorganizational elements in argumentative or persuasive discourse, wherethe computer-readable medium includes computer executable instructionswhich, when executed, cause the computer system to execute steps. Thesteps include receiving text that has been annotated, where theannotated text includes argumentative or persuasive discourse thatincludes claims and evidence and organizational elements configured toorganize the claims and evidence. Annotations of the annotated textdistinguish the organizational elements from the claims and evidence.The steps also include identifying a rule set or a feature set from theannotated text, where the rule set or the feature set include textualpatterns or word frequency features related to the organizationalelements of the annotated text. The steps further include building amodel based on the annotations and on the rule set or the feature set,where the model is configured to identify organizational elements in anew text. The model is applied to the new text.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a block diagram illustrating an example model building engineused to build one or more models, where the one or more models areconfigured to identify organizational elements in argumentative orpersuasive discourse.

FIG. 1B illustrates one or more example models being applied to a new,unlabeled (i.e., not annotated) text.

FIG. 2 depicts example annotated text used to build a model foridentifying organizational elements in argumentative or persuasivediscourse.

FIG. 3A illustrates example steps used in identifying a rule set fromannotated text, where the rule set is used in building a modelconfigured to identify organizational elements in argumentative orpersuasive discourse.

FIG. 3B depicts an example textual pattern (i.e., rule) that recognizesorganizational elements describing an author's position with respect toan opponent's opinion.

FIG. 4 is a flowchart including example steps used in building a modelto identify organizational elements based on a feature set identifiedfrom an annotated text.

FIG. 5 depicts an example system for applying a model to a new,unlabeled text, where the model is configured to identify organizationalelements in the new text.

FIG. 6 is a flowchart depicting operations of an example method foridentifying organizational elements in argumentative or persuasivediscourse.

FIGS. 7A, 7B, and 7C depict example systems for use in identifyingorganizational elements in argumentative or persuasive discourse.

DETAILED DESCRIPTION

FIG. 1A is a block diagram illustrating an example model building engine102 used to build one or more models 106, where the one or more models106 are configured to identify organizational elements in argumentativeor persuasive discourse. In FIG. 1A, the model building engine 102receives annotated text 104, where the annotated text 104 includesargumentative or persuasive discourse. The annotated text 104 mayinclude, for example, statements from a political debate, arguments froma legal brief or motion, or sentences from an essay rebutting anopponent's argument. The argumentative or persuasive discourse of theannotated text 104 includes not only language expressing claims andevidence but also language used to organize the claims and evidence(i.e., organizational elements). The annotated text 104 includesannotations identifying the claims and evidence 108 and annotationsidentifying the organizational elements 110 of the text 104. In oneexample, the annotations 108, 110 are carried out by humans experiencedin scoring argumentative or persuasive writing. In another example, theannotations 108, 110 are carried out by computer hardware or softwareconfigured to annotate text in this manner. Further, the annotated text104 may be annotated solely to distinguish the organizational elementsfrom the claims and evidence, or alternatively, the annotated text 104may be annotated to include these annotations 108, 110 and a variety ofothers (e.g., annotating the text to identify thesis statements andconclusion statements).

Annotations 108, 110 of the type illustrated in FIG. 1A reflect the factthat argumentative or persuasive discourse includes not only languageexpressing the claims and evidence (i.e., the “meat” of the argument)but also language used to organize the claims and evidence (i.e., the“shell” of the argument). Differentiating between the claims andevidence and the organizational elements may be used for a variety ofapplications, including those that focus on the content of arguments(e.g., relation extraction) and those that focus on the structure ofarguments (e.g., automated essay scoring). Detecting organizationalelements may also be a first step in parsing an argument to infer itsstructure. Further applications may be in the fields of politicalscience (e.g., to better understand political debates), informationextraction (e.g., to help a system focus on content rather thanorganization), and automated essay scoring (e.g., to analyze the qualityof a test-taker's argument).

The model building engine 102 receives the annotated text 104 andidentifies a rule set or a feature set 112 from the annotated text 104.The rule set or the feature set 112 may include textual patterns or wordfrequency features from the annotated text 104, where the textualpatterns or the word frequency features are related to theorganizational elements of the text 104. An example textual pattern(i.e., rule) may include parts of speech or word identifiers in aspecific sequence (e.g., “I [modal verb] [agree verb] with the [authornoun].”). An example word frequency feature may identify a relativefrequency of a word in a text corpus (e.g., for a word appearing in thetext 104, a word frequency feature may be related to the word's relativefrequency in the British National Corpus). The model building engine 102builds the one or more models 106 based on the annotations 108, 110 andon the rule set or feature set 112, where the one or more models 106 areconfigured to identify organizational elements in a new text (i.e., anunlabeled text that has not been annotated). The one or more models 106may function at the word level to identify whether each individual wordof the new text is an organizational element and may focus on explicitmarkers of organizational structure in arguments of the new text. Theone or more models 106 may include a rule-based model, a probabilisticsequence model, or a model that combines the rule-based and sequencemodel approaches.

FIG. 1B illustrates one or more example models 152 being applied to anew, unlabeled (i.e., not annotated) text 154. The one or more models152 may be similar to the one or more models 106 of FIG. 1A and may beconfigured to identify organizational elements 156 of the new text 154.The text 154 includes argumentative or persuasive discourse and mayinclude essays, text from political debates, and legal briefs 155, amongother types of argumentative or persuasive discourse. The organizationalelements 156 identified in the new text 154 may be processed for furtherapplications, including content analysis applications 158 and essayscoring applications 160.

In identifying the organizational elements 156, the one or more models152 may be applied in a variety of manners. In one example, a mostlikely sequence of labels given the words of the new text 154 isidentified using Viterbi decoding, where the sequence of labels includesbinary values indicating whether each word of the new text 154 is anorganizational element. In another example, a word of the new text 154is labeled as an organizational element if a marginal probabilityassociated with the word exceeds a threshold value.

FIG. 2 depicts example annotated text 200 used to build a model foridentifying organizational elements in argumentative or persuasivediscourse. As described above with respect to FIGS. 1A and 1B,persuasive or argumentative discourse includes organizational elementsand claims and evidence, where the organizational elements are words orsequences of words used to refer to the claims and evidence and toprovide an organizational framework for the argument. The organizationalelements may be used by a writer or speaker in a variety of ways,including (1) declaring one's own claims (e.g., “There is thepossibility that . . . ”); (2) restating an opponent's claims (e.g.,“The argument states that . . . ”); (3) evaluating an opponent's claims(e.g., “It may seem reasonable at first glance, but actually, there aresome logical mistakes in it.”); and (4) presenting evidence and relatingit to specific claims (e.g., “To illustrate my point, I will now givethe example of . . . ”).

In building a model to identify such organizational elements, annotatedtext is received, where the annotated text includes annotations used todistinguish organizational elements from claims and evidence of thetext. The annotated text 200 of FIG. 2 illustrates an example ofannotations that are received and used in building the model. Text 202is from an essay rebutting an opponent's statement that grizzly bearslived in a specific region of Canada. The underlined portion of the text202 is text that has been identified as including organizationalelements, and the portion that has not been underlined includes theclaims and evidence of the persuasive or argumentative discourse: “Theargument states that based on the result of the recent research, thereprobably were grizzly bears in Labrador. It may seem reasonable at firstglance, but actually, there are some logical mistakes in it . . . .There is a possibility that they were a third kind of bear apart fromblack and grizzly bears. Also, the explorer accounts were recorded inthe nineteenth century, which was more than 100 years ago . . . . Insum, the conclusion of this argument is not reasonable since the accountand the research are not convincing enough.”

Text 204 includes text from a political debate, with the underliningagain being an annotation indicating that the underlined text includesan organizational element: “But the point is—the point is, we havefinally seen Republicans and Democrats sitting down and negotiatingtogether . . . . And one of the things I think we have to do is makesure that college is affordable.”

The text 200 may be annotated by a human (e.g., by individualsexperienced in scoring persuasive writing, with or without formalguidelines being provided by another) or by a computer (e.g., hardwareor software configured to annotate text to distinguish organizationalelements from claims and evidence in persuasive or argumentativediscourse). In one example, annotations in the annotated text 200 aremade at the word level, such that a determination is made as to whethereach individual word is an organizational element or not.

FIG. 3A illustrates example steps 300 used in identifying a rule setfrom annotated text, where the rule set is used in building a modelconfigured to identify organizational elements in argumentative orpersuasive discourse. As described above with respect to FIG. 1A, amodel-building engine may receive annotated text and identify a rule setfrom the annotated text, where the rule set includes textual patterns(i.e., rules) related to organizational elements of the annotated text.The model built based on the rule set and the textual patterns may thusbe used in implementing a rule-based system for identifyingorganizational elements in persuasive or argumentative discourse.

At 301, annotated text is received. Annotations of the annotated textdistinguish organizational elements from claims and evidence of thetext. At 302, an n-gram that includes organizational elements isidentified in the annotated text. The n-gram is a contiguous sequence ofn items (e.g., n words) from the annotated text that is identified basedon the annotations. An example n-gram including organizational elementsis illustrated at 306: “I cannot totally agree with the speaker . . . .”At 304, a textual pattern that describes the n-gram is determined. Thetextual pattern is an abstraction of the n-gram that includes parts ofspeech or word identifiers in a specific sequence. An example textualpattern that describes the example n-gram at 306 is illustrated at 308:“I [modal verb] [adverb] [agree verb] with the [author noun] . . . . ”The textual pattern may function as a “rule,” where sequences of textthat satisfy the constraints of the rule may be determined to includeorganizational elements. Although the steps 300 describe identifying asingle n-gram and a single textual pattern, in practice, lists offrequent n-grams may be identified from the annotated text, with n=1, 2,. . . , 9, for example. Multiple textual patterns (i.e., rules) may becomputed to recognize the organizational elements present in the listsof frequent n-grams. Because the textual patterns are abstractions ofthe n-grams, a single textual pattern may describe more than one n-gramin the lists of n-grams.

FIG. 3B depicts an example textual pattern (i.e., rule) 360 thatrecognizes organizational elements describing an author's position withrespect to an opponent's opinion. The textual pattern 360 is as follows:“I [modal verb] [adverb] [agree verb] with the [author noun] . . . . ”Example modal verbs 352 include “do,” “don't,” “can,” “cannot,” “will,”and “would.” Example adverbs 354 include “strongly,” “totally,”“fundamentally,” and “vehemently.” Example agree verbs 356 include“disagree,” “agree,” and “concur.” Example author nouns 358 include“writer,” “author,” and “speaker.” In one example of the rule-basedsystem, twenty-five (25) textual patterns are identified from annotatedtext (i.e., annotated essays, where the essays were produced bytest-takers of a standardized test). In a model, the textual patterns ofthe rule set are applied to new text, where sequences of the new textthat satisfy the constraints of the textual patterns may be determinedto include organizational elements.

FIG. 4 is a flowchart 400 including example steps used in building amodel to identify organizational elements based on a feature setidentified from an annotated text. At 402, an annotated text isreceived. The annotated text may be a labeled dataset of N examples (w,y) indexed by i, containing sequences of words w^(i) and sequences oflabels y^(i), with individual words and labels indexed by j. y^(i) is asequence of binary values, indicating whether each word w_(j) ^(i) inthe sequence is an organizational element (y_(j) ^(i)=1) or not (y_(j)^(i)=0).

At 404, a feature set is identified from the annotated text, where thefeature set includes word frequency features or other features. In oneexample, the feature set may include a number of feature values, wherethe feature values for the jth word and label pair of the annotated textare as follows:

(1) A first relative frequency of w_(j) ^(i) in a corpus (e.g., theBritish National Corpus);

(2) A second relative frequency of w_(j) ^(i) in a set of essays (e.g.,100,000 essays);

(3) Binary features indicating whether the first and second relativefrequencies meet or exceed one or more thresholds (e.g., thresholds of10^({−6, −5, −4, −3}));

(4) A number of different essay prompts in which w_(j) ^(i) appeared;

(5) Binary features indicating whether the number of different essayprompts meets or exceeds one or more thresholds (e.g., thresholds of{0.25, 0.50, 0.75});

(6) A binary feature with value 1 if w_(j) ^(i) consists of only lettersa-z, and 0 otherwise (i.e., a feature to distinguish punctuation andnumbers from other tokens);

(7) A binary feature with value 1 if a rule-based system (e.g., therule-based system described with respect to FIGS. 3A and 3B) predictsthat w_(j) ^(i) is an organizational element, and with a value of 0otherwise;

(8) A binary feature with value 1 if the rule-based system predicts thatw_(j-1) ^(i) is an organizational element, and 0 otherwise;

(9) Two binary features for whether or not the current token was thefirst or last in the sentence, respectively; and

(10) Four binary features for the possible transitions between previousand current labels (y_(j) ^(i) and y_(j-1) ^(i) respectively).

Feature values (1)-(10) are summarized at 452 of FIG. 4. In building themodel, f is a feature function that takes pairs of word and labelsequences from the annotated text and returns a vector of featurevalues. To compute the values of f for a sequence of words and labels,the feature values (1)-(10) above are summed over all elements of asequence. Building a model based on features (1)-(10) combines arule-based system and a probabilistic sequence model, as certain of thefeatures utilize the rule-based system within the context of theprobabilistic sequence model. In other examples, certain of features(1)-(10) are not used, and as such, a model may be based on a singlefeature of features (1)-(10) or multiple of the features in variouscombinations. Further, in certain example systems, the features of(1)-(10) that are based on the rule-based system are not used.

At 406, a parameter vector θ that maximizes the following objectivefunction is determined:

${{L\left( {\left. \theta \middle| w \right.,y} \right)} = {\sum\limits_{i = 1}^{N}{p_{\theta}\left( y^{i} \middle| w^{i} \right)}}},{{L\left( {\left. \theta \middle| w \right.,y} \right)} = {\sum\limits_{i = 1}^{N}\left( {{\theta^{T}{f\left( {w^{i},y^{i}} \right)}} - {\log \; Z}} \right)}},$

where a normalization constant Z sums over all possible label sequences.The vector of feature values f is equal in dimensions to the number ofparameters in θ. The vector θ may be viewed as including weightingfactors for the values of vector f.

The parameter vector θ and the function p_(θ) are used in implementingthe model configured to identify organizational elements in a new text.Building the model in this manner may be used in implementing asupervised, probabilistic sequence model based on conditional randomfields (CRFs). Determining the parameter vector θ and the function p_(θ)is thus used to define a probability distribution based on the rule setor the feature set.

FIG. 5 depicts an example system 500 for applying a model to a new,unlabeled text 502, where the model is configured to identifyorganizational elements in the new text 502. The system 500 is thusconfigured to make predictions about the new, unlabeled text 502, wherethe predictions may include binary values indicating whether each wordof the new text 502 is an organizational element. In FIG. 5, the newtext 502 is received by a text processing unit 506. The text processingunit 506 may be configured to perform a variety of text processingapplications on the new text 502, including tokenization, syntacticparsing, and stemming 508, among others. The text processing is used toprepare the new text 502 for further processing in one or more models510.

The one or more models 510 may include multiple models that utilizedifferent model types. The one or more models 510 may include arule-based model (e.g., as described above with respect to FIGS. 3A and3B), a supervised sequence model (e.g., as described above with respectto FIG. 4), or a combination of these models or others (e.g., thesupervised sequence model of FIG. 4 that includes features determinedbased on the rule-based model of FIGS. 3A and 3B).

Various of the one or more models 510 may also allow a particular modelto be applied to the new text 502 in a variety of methods 511 (e.g.,methods using most likely sequences determined with Viterbi decoding andmethods based on marginal probabilities, as described in further detailbelow). For example, multiple of the one or more models 510 mayimplement the supervised sequence model, but these models may bedistinguishable in their application of the supervised sequence model.When making predictions ŷ^(i) about a sentence in the new text 502, onemethod of applying the supervised sequence model is to find a mostlikely sequence of labels y given the words w^(i) of the new text 502using Viterbi decoding:

${{\hat{y}}^{i} = {\underset{y}{argmax}\; {p_{\theta}\left( y \middle| w^{i} \right)}}},$

where y ^(i) includes predictions about the words w^(i) of the new text502, the predictions being binary values indicating whether each of thewords w^(i) are organizational elements, and p_(θ) is a functiondetermined based on a feature set identified from annotated trainingdata (e.g., the function p_(θ) defined above with respect to FIG. 4). Inan example, variables of the preceding equation are defined consistentlywith the variables defined above with respect to FIG. 4. The p_(θ)function may vary based on the features used in the feature set (e.g.,the feature set may include features based on the rule-based system ornot). In one example, an l₂ penalty on the magnitude of θ in thefunction p_(θ) may be implemented.

An alternative method of applying the supervised sequence model to thenew text 502 labels each word as an organizational element if the sum ofthe probabilities of all paths in which the word was labeled as anorganizational element (i.e., the marginal probability) exceeds somethreshold λ. Words are not labeled as organizational elements otherwise(i.e., they are identified as being claims or evidence). Specifically,an individual word w_(j) ^(i) is labeled as an organizational element(i.e., y_(j) ^(i)=1) according to the following equation:

${\hat{y}}_{j}^{i} = {1\left( {\left( {\sum\limits_{y}^{\;}{{p_{\theta}\left( y \middle| w^{i} \right)}1\left( {y_{j} = 1} \right)}} \right) \geq \lambda} \right)}$

where ŷ_(j) ^(i) is a prediction for the word w_(i) ^(j), where the wordw_(j) ^(i) is part of a sequence w^(i) of the new text 502, theprediction is a binary value indicating whether the word w_(j) ^(i) isan organizational element, and p_(θ) is a function determined based on afeature set identified from annotated training data (e.g., the functionp_(θ) defined above with respect to FIG. 4). In an example, variables ofthe preceding equation are defined consistently with the variablesdefined above with respect to FIG. 4. The threshold λ may be tuned usingtraining data.

Based on the one or more models 510 selected and on the particularmethod of applying the selected model, the system 500 identifiesorganizational elements 512 (i.e., the “shell” used to organize the“meat” of the arguments of text 502). The organizational elements 512may be processed for further applications, including content analysisapplications and essay scoring applications. In one example application,the organizational elements 512 are received by a score determinationmodule 514 and used to produce a score 516 associated with the text 502.

In certain examples, a performance of the one or more models 510 may beevaluated by comparing results of the one or more models 510 (i.e.,annotations produced by the one or more models 510 identifying theorganizational elements 512) to annotations produced by humans. In oneexample, performance of the one or more models 510 is measured at theword token-level using metrics including precision, recall, or the F₁measure. For example, for the precision metric, a proportion of tokenspredicted to be organizational elements by the one or more models 510that were also labeled as organizational elements by humans isdetermined.

FIG. 6 is a flowchart 600 depicting operations of an example method foridentifying organizational elements in argumentative or persuasivediscourse. At 602, text that has been annotated to distinguishorganizational elements from claims and evidence are received. The textincludes argumentative or persuasive discourse and includesorganizational elements and claims and evidence. At 604, a rule set or afeature set is identified from the annotated text. The rule set orfeature set includes textual patterns or word frequency features relatedto the organizational elements of the annotated text. At 606, a model isbuilt based on the annotations and the rule set or feature set. Themodel is configured to identify organizational elements in a new text.At 608, the model is applied to new text to identify organizationalelements in the new text.

FIGS. 7A, 7B, and 7C depict example systems for use in identifyingorganizational elements in argumentative or persuasive discourse. Forexample, FIG. 7A depicts an exemplary system 700 that includes astandalone computer architecture where a processing system 702 (e.g.,one or more computer processors located in a given computer or inmultiple computers that may be separate and distinct from one another)includes one or more models 704 being executed on it. The processingsystem 702 has access to a computer-readable memory 706 in addition toone or more data stores 708. The one or more data stores 708 may includeannotated text 710 as well as rule sets 712. The processing system 702may be a distributed parallel computing environment, which may be usedto handle very large-scale data sets.

FIG. 7B depicts a system 720 that includes a client-server architecture.One or more user PCs 722 access one or more servers 724 running one ormore models 726 on a processing system 727 via one or more networks 728.The one or more servers 724 may access a computer-readable memory 730 aswell as one or more data stores 732. The one or more data stores 732 maycontain annotated text 734 as well as rule sets 736.

FIG. 7C shows a block diagram of exemplary hardware for a standalonecomputer architecture 750, such as the architecture depicted in FIG. 7Athat may be used to contain and/or implement the program instructions ofsystem embodiments of the present disclosure. A bus 752 may serve as theinformation highway interconnecting the other illustrated components ofthe hardware. A processing system 754 labeled CPU (central processingunit) (e.g., one or more computer processors at a given computer or atmultiple computers), may perform calculations and logic operationsrequired to execute a program. A non-transitory processor-readablestorage medium, such as read only memory (ROM) 756 and random accessmemory (RAM) 758, may be in communication with the processing system 754and may contain one or more programming instructions for performing themethod of identifying organizational elements in argumentative orpersuasive discourse. Optionally, program instructions may be stored ona non-transitory computer-readable storage medium such as a magneticdisk, optical disk, recordable memory device, flash memory, or otherphysical storage medium.

In FIGS. 7A, 7B, and 7C, computer readable memories 706, 730, 756, 758or data stores 708, 732, 762, 764, 766 may include one or more datastructures for storing and associating various data used in the examplesystems for identifying organizational elements in argumentative orpersuasive discourse. For example, a data structure stored in any of theaforementioned locations may be used to associate organizationalelements and claims and evidence with a given annotated text. As anotherexample, a data structure may be used to relate organizational elementsidentified in an unlabeled text with the unlabeled text. Further, theorganizational elements and the unlabeled text may be associated with ascore generated based on the organizational elements and other aspectsof the unlabeled text. Other aspects of the example systems foridentifying organizational elements in argumentative or persuasivediscourse may be stored and associated in the one or more datastructures (e.g., n-grams, textual patterns, and features identified inan annotated text).

A disk controller 760 interfaces one or more optional disk drives to thesystem bus 752. These disk drives may be external or internal floppydisk drives such as 762, external or internal CD-ROM, CD-R, CD-RW or DVDdrives such as 764, or external or internal hard drives 766. Asindicated previously, these various disk drives and disk controllers areoptional devices.

Each of the element managers, real-time data buffer, conveyors, fileinput processor, database index shared access memory loader, referencedata buffer and data managers may include a software application storedin one or more of the disk drives connected to the disk controller 760,the ROM 756 and/or the RAM 758. The processor 754 may access one or morecomponents as required.

A display interface 768 may permit information from the bus 752 to bedisplayed on a display 770 in audio, graphic, or alphanumeric format.Communication with external devices may optionally occur using variouscommunication ports 772.

In addition to these computer-type components, the hardware may alsoinclude data input devices, such as a keyboard 773, or other inputdevice 774, such as a microphone, remote control, pointer, mouse and/orjoystick.

Additionally, the methods and systems described herein may beimplemented on many different types of processing devices by programcode comprising program instructions that are executable by the deviceprocessing subsystem. The software program instructions may includesource code, object code, machine code, or any other stored data that isoperable to cause a processing system to perform the methods andoperations described herein and may be provided in any suitable languagesuch as C, C++, JAVA, for example, or any other suitable programminglanguage. Other implementations may also be used, however, such asfirmware or even appropriately designed hardware configured to carry outthe methods and systems described herein.

The systems' and methods' data (e.g., associations, mappings, datainput, data output, intermediate data results, final data results, etc.)may be stored and implemented in one or more different types ofcomputer-implemented data stores, such as different types of storagedevices and programming constructs (e.g., RAM, ROM, Flash memory, flatfiles, databases, programming data structures, programming variables,IF-THEN (or similar type) statement constructs, etc.). It is noted thatdata structures describe formats for use in organizing and storing datain databases, programs, memory, or other computer-readable media for useby a computer program.

The computer components, software modules, functions, data stores anddata structures described herein may be connected directly or indirectlyto each other in order to allow the flow of data needed for theiroperations. It is also noted that a module or processor includes but isnot limited to a unit of code that performs a software operation, andcan be implemented for example as a subroutine unit of code, or as asoftware function unit of code, or as an object (as in anobject-oriented paradigm), or as an applet, or in a computer scriptlanguage, or as another type of computer code. The software componentsand/or functionality may be located on a single computer or distributedacross multiple computers depending upon the situation at hand.

While the disclosure has been described in detail and with reference tospecific embodiments thereof, it will be apparent to one skilled in theart that various changes and modifications can be made therein withoutdeparting from the spirit and scope of the embodiments. Thus, it isintended that the present disclosure cover the modifications andvariations of this disclosure provided they come within the scope of theappended claims and their equivalents.

It is claimed:
 1. A computer-implemented method for identifyingorganizational elements in argumentative or persuasive discourse, themethod comprising: receiving text that has been annotated, the annotatedtext including argumentative or persuasive discourse that includes:claims and evidence, and organizational elements configured to organizethe claims and evidence, wherein annotations of the annotated textdistinguish the organizational elements from the claims and evidence;identifying a rule set or a feature set from the annotated text, therule set or the feature set including textual patterns or word frequencyfeatures related to the organizational elements of the annotated text;building a model based on the annotations and the rule set or thefeature set, the model being configured to identify organizationalelements in a new text; and applying the model to the new text.
 2. Themethod of claim 1, further comprising: receiving the annotated textincluding the organizational elements, wherein the organizationalelements refer to the claims and evidence and provide an organizationalframework for the argumentative or persuasive discourse, and wherein theorganizational elements: declare claims of a writer, speaker, or author;restate claims of an opponent of the writer, speaker, or author;evaluate the claims of the opponent; or present evidence and relate theevidence to particular claims.
 3. The method of claim 1, furthercomprising: building the model based on the feature set, wherein thefeature set includes features based on the rule set and the textualpatterns.
 4. The method of claim 1, further comprising: identifying therule set from the annotated text, wherein the identifying includes:identifying an n-gram in the annotated text that includes organizationalelements, wherein the n-gram is a contiguous sequence of n items fromthe annotated text that is identified based on the annotations;determining a textual pattern that describes the n-gram, wherein thetextual pattern is an abstraction of the n-gram that includes parts ofspeech or word identifiers in a specific sequence; and storing thetextual pattern in the rule set.
 5. The method of claim 4, furthercomprising: determining the textual pattern that describes the n-gram,wherein the textual pattern describes more than one n-gram in theannotated text.
 6. The method of claim 4, further comprising:determining the textual pattern that describes the n-gram, wherein theparts of speech or word identifiers include a modal verb, an agree verb,an author noun, a verb, a noun, a pronoun, an adjective, an adverb, aproposition, a conjunction, or an interjection.
 7. The method of claim1, further comprising: building the model based on the annotations andthe rule set or the feature set, wherein the building includesmaximizing an objective function, the objective function including termscorresponding to the textual patterns or word frequency features.
 8. Themethod of claim 7, further comprising: building the model based on theannotations and the rule set or the feature set, wherein the annotatedtext includes N examples (w, y) indexed by i, including sequences ofwords w^(i) and sequences of labels y^(i), with individual words andlabels indexed by j, wherein y^(i) is a sequence of binary valuesindicating whether each word w_(j) ^(i) in the sequence is anorganizational element, and wherein the building includes: determining aparameter vector θ that maximizes the objective function, wherein theobjective function is defined as:${{L\left( {\left. \theta \middle| w \right.,y} \right)} = {\sum\limits_{i = 1}^{N}{p_{\theta}\left( y^{i} \middle| w^{i} \right)}}},$where p_(θ)(y^(i)|w^(i)) is defined as:p _(θ)(y ^(i) |w ^(i))=(θ^(T) f(w ^(i) ,y ^(i))−log Z), where Z is anormalization constant that sums over all possible label sequences, andf is a feature function that takes pairs of word and label sequences andreturns a vector of feature values equal in dimensions to the number ofparameters in θ.
 9. The method of claim 1, further comprising:identifying the feature set from the annotated text, wherein the wordfrequency features include: a relative frequency for a first word of theannotated text in a text corpus; a first binary value identifyingwhether the relative frequency meets or exceeds a first threshold; anumber of different essay prompts in which the first word has appeared;or a second binary value identifying whether the number of differentessay prompts meets or exceeds a second threshold.
 10. The method ofclaim 9, further comprising: identifying the feature set from theannotated text, wherein features of the feature set include: a thirdbinary value identifying whether the first word includes only letters; afourth binary value identifying whether a rule-based system predicts thefirst word as being an organizational element; a fifth binary valueidentifying whether the rule-based system predicts a second word asbeing an organizational element, wherein the second word immediatelyprecedes the first word in the annotated text; a sixth binary valueidentifying whether the first word occurs at the beginning of a asentence; a seventh binary value identifying whether the first wordoccurs at the end of the sentence; or an eighth binary value identifyinga transition between a label applied to the first word and a labelapplied to the second word, wherein the first and the second labelsindicate whether the first and the second words, respectively, areorganizational elements.
 11. The method of claim 1, further comprising:applying the model to the new text, wherein the applying includesfinding a most likely sequence of labels y given a sequence of wordsw^(i) of the new text, wherein the most likely sequence of labels y is avector including binary values indicating whether words of the sequenceof words w^(i) are organizational elements, and wherein the most likelysequence of labels y is determined using Viterbi decoding.
 12. Themethod of claim 11, further comprising: determining the most likelysequence of labels y using the Viterbi decoding according to a followingequation:${{\hat{y}}^{i} = {\underset{y}{argmax}\; {p_{\theta}\left( y \middle| w^{i} \right)}}},$where ŷ^(i) is a sequence of predictions about the sequence of wordsw^(i) of the new text, the predictions being binary values indicatingwhether the sequence of words w^(i) includes organizational elements,and where p_(θ) is a function based on the rule set or the feature setfrom the annotated text.
 13. The method of claim 1, further comprising:applying the model to the new text, wherein the applying includesdetermining whether a marginal probability for a word w_(j) ^(i) exceedsa threshold λ.
 14. The method of claim 13, further comprising:determining whether the marginal probability for the word w_(j) ^(i)exceeds the threshold λ to determine a binary value ŷ_(j) ^(i), whereinthe binary value ŷ_(j) ^(i) indicates whether the word w_(j) ^(i) is anorganizational element, and wherein the binary value ŷ_(j) ^(i) isdetermined according to an equation:${\hat{y}}_{j}^{i} = {1\left( {\left( {\sum\limits_{y}^{\;}{{p_{\theta}\left( y \middle| w^{i} \right)}1\left( {y_{j} = 1} \right)}} \right) \geq \lambda} \right)}$where p_(θ) is a function based on the rule set or the feature set fromthe annotated text, and where w^(i) is a sequence of words withindividual words and labels indexed by j.
 15. The method of claim 1,further comprising: evaluating the model, the evaluating including:applying the model to the new text, wherein the applying includesgenerating a set of predictions using the model, and wherein thepredictions identify organizational elements of the new text; comparingthe predictions to labels generated by humans, wherein the labelsidentify organizational elements of the new text; and based on thecomparing, evaluating the model using precision, recall, or F₁ measures.16. A system for identifying organizational elements in argumentative orpersuasive discourse, the system comprising: a data processor; andcomputer-readable memory in communication with the data processorencoded with instructions for commanding the data processor to executesteps comprising: receiving text that has been annotated, the annotatedtext including argumentative or persuasive discourse that includes:claims and evidence, and organizational elements configured to organizethe claims and evidence, wherein annotations of the annotated textdistinguish the organizational elements from the claims and evidence;identifying a rule set or a feature set from the annotated text, therule set or the feature set including textual patterns or word frequencyfeatures related to the organizational elements of the annotated text;building a model based on the annotations and on the rule set or thefeature set, the model being configured to identify organizationalelements in a new text; and applying the model to the new text.
 17. Thesystem of claim 16, wherein the steps further comprise: applying themodel to the new text, wherein the model is applied to generate a scorefor the new text, and wherein the new text is an essay.
 18. The systemof claim 17, wherein the model includes a probability distribution basedon the rule set or the feature set.
 19. The system of claim 16, whereinthe organizational elements: declare claims of a writer, speaker, orauthor; restate claims of an opponent of the writer, speaker, or author;evaluate the claims of the opponent; or present evidence and relate theevidence to particular claims.
 20. A non-transitory computer-readablestorage medium for identifying organizational elements in argumentativeor persuasive discourse, the computer-readable medium comprisingcomputer executable instructions which, when executed, cause thecomputer system to execute steps comprising: receiving text that hasbeen annotated, the annotated text including argumentative or persuasivediscourse that includes: claims and evidence, and organizationalelements configured to organize the claims and evidence, whereinannotations of the annotated text distinguish the organizationalelements from the claims and evidence; identifying a rule set or afeature set from the annotated text, the rule set or the feature setincluding textual patterns or word frequency features related to theorganizational elements of the annotated text; building a model based onthe annotations and on the rule set or the feature set, the model beingconfigured to identify organizational elements in a new text; andapplying the model to the new text.